Subjective evaluation of spoken dialogue systems using SER VQUAL method
نویسندگان
چکیده
There is demand for subjective metrics in spoken dialogue system evaluation. SERVQUAL is a service quality evaluation method developed by marketing academics. It produces a subjective measure of the gap between expectations and perceptions in five service quality dimensions common for all services. We present how the method was applied to spoken dialogue system evaluation. In order to improve the suitability of the original method, we modified the test questionnaire and the test process. We demonstrate how the modified method was successfully used in an evaluation of a telephone-based e-mail application. The evaluation gave us directions for further development of the system. In addition, we found some interesting phenomena, such as the variation between genders. We present how the method can be further improved, for example, by dividing the questionnaire into two parts.
منابع مشابه
Quality of telephone based spoken dialogue systems pdf
Quality of Telephone-Based Spoken Dialogue Systems. Quality of Human-Machine Interaction over the Phone.Quality prediction models for telephone-based spoken dialogue systems. Perform to guarantee an acceptable overall quality for the user. In order to.action HHI as one reference for telephonebased humanmachine interaction HMI. The quality of interactions with spoken dialogue systems is difficul...
متن کاملAdaptive Information Presentation for Spoken Dialogue Systems: Evaluation with real users
We present evaluation results with human subjects for a novel data-driven approach to Natural Language Generation in spoken dialogue systems. We evaluate a trained Information Presentation (IP) strategy in a deployed tourist-information spoken dialogue system. The IP problem is formulated as statistical decision making under uncertainty using Reinforcement Learning, where both content planning ...
متن کاملDialogue Strategy Redesign with Reliability Measures
This paper reports on user evaluation for a spoken dialogue system and the subsequent redesign of the dialogue strategy. An automatic telephone directory system developed at Philips Research, PADIS, was evaluated using both objective metrics and subjective evaluation techniques. Duration and the occurrence of misunderstanding leading tòdead-ends', were found to be the key factors that kept user...
متن کاملITU-T Rec. P.851 (11/2003) Subjective quality evaluation of telephone services based on spoken dialogue systems
متن کامل
A Data-driven Model for Timing Feedback in a Map Task Dialogue System
We present a data-driven model for detecting suitable response locations in the user’s speech. The model has been trained on human–machine dialogue data and implemented and tested in a spoken dialogue system that can perform the Map Task with users. To our knowledge, this is the first example of a dialogue system that uses automatically extracted syntactic, prosodic and contextual features for ...
متن کامل